Skip to content

feat: add account_advisory cleanup to advisory cleanup job - #2290

Open
katarinazaprazna wants to merge 2 commits into
RedHatInsights:masterfrom
katarinazaprazna:parallel-advisory-cleanup
Open

feat: add account_advisory cleanup to advisory cleanup job#2290
katarinazaprazna wants to merge 2 commits into
RedHatInsights:masterfrom
katarinazaprazna:parallel-advisory-cleanup

Conversation

@katarinazaprazna

@katarinazaprazna katarinazaprazna commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Extend the advisory cleanup job to also clean zero-count rows from the new account_advisory table, running both cleanups in parallel until the legacy table is dropped

  • Add row-count logging to cleanups for observability

  • Prevent deleting advisory_metadata that still has rows in the new table. However, account_advisory fans out across 32 hash partitions per candidate, bounded by LIMIT 1000. Do you think we should validate this on stage before merging, or are we good?

Follow-up

  • PUT /clean-advisory-account-data admin API endpoint only cleans the legacy table. Update to also clean account_advisory (or rename/replace when legacy table is dropped)

Secure Coding Practices Checklist GitHub Link

Secure Coding Checklist

  • Input Validation
  • Output Encoding
  • Authentication and Password Management
  • Session Management
  • Access Control
  • Cryptographic Practices
  • Error Handling and Logging
  • Data Protection
  • Communication Security
  • System Configuration
  • Database Security
  • File Management
  • Memory Management
  • General Coding Practices

Summary by Sourcery

Add a new job that concurrently cleans legacy and new account advisory data while ensuring advisory metadata is only removed when unused by both tables.

New Features:

  • Introduce a consolidated clean_account_advisory job that runs cleanup for both advisory_account_data and account_advisory tables in parallel.

Bug Fixes:

  • Prevent deletion of advisory_metadata rows that are still referenced by the new account_advisory table.

Enhancements:

  • Add logging of deleted row counts for advisory_account_data and account_advisory cleanups for better observability.

Build:

  • Rename and reconfigure the scheduled cleanup cronjob and related parameters to use the new clean_account_advisory job.

Tests:

  • Add a regression test to ensure advisories referenced by account_advisory are not deleted by the unused advisory cleanup job.

@sourcery-ai

sourcery-ai Bot commented Aug 3, 2026

Copy link
Copy Markdown

Reviewer's Guide

Adds a new parallel cleanup job for legacy advisory_account_data and the new account_advisory table, updates the unused-advisories deletion query to respect the new table, wires a new job entrypoint/cron config, and adds a DB test ensuring metadata referenced by account_advisory is preserved.

File-Level Changes

Change Details Files
Introduce a unified cleanup job that runs legacy advisory_account_data and new account_advisory deletions in parallel with row-count logging.
  • Add RunCleanAccountAdvisory entrypoint that configures the app, installs cancel handling, runs both cleanups concurrently, and logs completion.
  • Implement CleanAdvisoryAccountData to delete rows with zero applicable/installable systems from advisory_account_data in a transaction and log deleted row count.
  • Implement CleanAccountAdvisory to delete rows with zero applicable/installable systems from account_advisory in a transaction and log deleted row count.
  • Remove the old clean_advisory_account_data job implementation now that it is folded into the new unified job.
tasks/cleaning/clean_account_advisory.go
tasks/cleaning/clean_advisory_account_data.go
Update unused advisory deletion to account for references from the new account_advisory table and add a regression test.
  • Extend deleteUnusedAdvisories subquery to also check for NOT EXISTS rows in account_advisory for the same advisory_id, preventing deletion of still-referenced metadata.
  • Add a DB-backed test that creates AdvisoryMetadata plus a referencing AccountAdvisory row and asserts deleteUnusedAdvisories does not remove the metadata.
  • Use uuid.MustParse to set a deterministic WorkspaceID in the new test and clean up inserted rows after the test.
tasks/cleaning/clean_unused_data.go
tasks/cleaning/clean_unused_data_test.go
Wire the new cleanup job name into the application entrypoint and clowdapp cron configuration.
  • Change main.go job switch to route clean_account_advisory to cleaning.RunCleanAccountAdvisory instead of the removed clean_advisory_account_data.
  • Rename the clowdapp cronjob from clean-advisory-account-data to clean-account-advisory and update its job command accordingly.
  • Rename related environment parameters from CLEAN_AAD_* to CLEAN_AA_* to match the new job name and semantics.
main.go
deploy/clowdapp.yaml

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@codecov-commenter

codecov-commenter commented Aug 3, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 2.22222% with 44 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.97%. Comparing base (f7d5d18) to head (82efcf3).

Files with missing lines Patch % Lines
tasks/cleaning/clean_account_advisory.go 0.00% 42 Missing ⚠️
main.go 0.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2290      +/-   ##
==========================================
- Coverage   59.12%   58.97%   -0.16%     
==========================================
  Files         149      149              
  Lines        9549     9575      +26     
==========================================
+ Hits         5646     5647       +1     
- Misses       3311     3336      +25     
  Partials      592      592              
Flag Coverage Δ
unittests 58.97% <2.22%> (-0.16%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@katarinazaprazna
katarinazaprazna force-pushed the parallel-advisory-cleanup branch from d447030 to 5c64f33 Compare August 3, 2026 00:30
@katarinazaprazna
katarinazaprazna marked this pull request as ready for review August 3, 2026 11:06
@katarinazaprazna
katarinazaprazna requested a review from a team as a code owner August 3, 2026 11:06

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've left some high level feedback:

  • RunCleanAccountAdvisory always logs task performed successfully even if one or both cleaners fail; consider aggregating error results and surfacing a non-success log or exit status when any delete operation returns an error.
  • CleanAdvisoryAccountData and CleanAccountAdvisory share nearly identical transaction and logging logic; consider extracting a small helper that accepts the model and condition to reduce duplication and keep the behavior consistent across both tables.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- RunCleanAccountAdvisory always logs `task performed successfully` even if one or both cleaners fail; consider aggregating error results and surfacing a non-success log or exit status when any delete operation returns an error.
- CleanAdvisoryAccountData and CleanAccountAdvisory share nearly identical transaction and logging logic; consider extracting a small helper that accepts the model and condition to reduce duplication and keep the behavior consistent across both tables.

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@katarinazaprazna
katarinazaprazna force-pushed the parallel-advisory-cleanup branch from 5c64f33 to 3a644c6 Compare August 3, 2026 11:08
@katarinazaprazna
katarinazaprazna marked this pull request as draft August 3, 2026 11:11
@katarinazaprazna
katarinazaprazna force-pushed the parallel-advisory-cleanup branch from 3a644c6 to 092f42d Compare August 3, 2026 11:18
Extend the advisory cleanup job to also clean zero-count rows from the
new account_advisory table, running both cleanups in parallel. The
legacy advisory_account_data cleanup runs until that table is dropped.
@katarinazaprazna
katarinazaprazna force-pushed the parallel-advisory-cleanup branch from 092f42d to 27e51dd Compare August 3, 2026 12:08
@katarinazaprazna
katarinazaprazna marked this pull request as ready for review August 3, 2026 12:09

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 2 issues, and left some high level feedback:

  • The CleanAdvisoryAccountData and CleanAccountAdvisory functions are nearly identical; consider extracting a shared helper that takes the model type/table name to reduce duplication and keep the cleanup logic consistent in one place.
  • In both cleanup functions you call Begin() with defer tx.Rollback() and then tx.Commit() without checking its error; consider either using the base DB without an explicit transaction for this single-statement delete, or handling Commit() errors explicitly and avoiding a deferred rollback after a successful commit.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- The `CleanAdvisoryAccountData` and `CleanAccountAdvisory` functions are nearly identical; consider extracting a shared helper that takes the model type/table name to reduce duplication and keep the cleanup logic consistent in one place.
- In both cleanup functions you call `Begin()` with `defer tx.Rollback()` and then `tx.Commit()` without checking its error; consider either using the base DB without an explicit transaction for this single-statement delete, or handling `Commit()` errors explicitly and avoiding a deferred rollback after a successful commit.

## Individual Comments

### Comment 1
<location path="tasks/cleaning/clean_account_advisory.go" line_range="11-20" />
<code_context>
+	"sync"
+)
+
+func RunCleanAccountAdvisory() {
+	tasks.HandleContextCancel(tasks.WaitAndExit)
+	core.ConfigureApp()
+	defer utils.LogPanics(true)
+
+	var wg sync.WaitGroup
+	wg.Add(2)
+
+	go func() {
+		defer wg.Done()
+		utils.LogInfo("Deleting advisory rows with 0 applicable/installable systems from advisory_account_data")
+		if err := CleanAdvisoryAccountData(); err != nil {
+			utils.LogError("err", err, "Cleaning advisory_account_data")
+		}
+	}()
+
+	go func() {
+		defer wg.Done()
+		utils.LogInfo("Deleting advisory rows with 0 applicable/installable systems from account_advisory")
+		if err := CleanAccountAdvisory(); err != nil {
+			utils.LogError("err", err, "Cleaning account_advisory")
+		}
+	}()
+
+	wg.Wait()
+	utils.LogInfo("RunCleanAccountAdvisory task performed successfully")
+}
+
</code_context>
<issue_to_address>
**issue (bug_risk):** Consider surfacing failures from the two cleanup routines instead of always logging success

Right now, errors from `CleanAdvisoryAccountData` / `CleanAccountAdvisory` are only logged inside the goroutines, but `RunCleanAccountAdvisory` still logs a blanket success message. This can misrepresent job status and interfere with alerting. Consider aggregating errors (e.g., via a channel or shared error with synchronization) so that you can emit a failure log when any cleanup fails and optionally exit with a non-zero status for the job runner.
</issue_to_address>

### Comment 2
<location path="tasks/cleaning/clean_account_advisory.go" line_range="39-48" />
<code_context>
+	utils.LogInfo("RunCleanAccountAdvisory task performed successfully")
+}
+
+func CleanAdvisoryAccountData() error {
+	tx := tasks.CancelableDB().Begin()
+	defer tx.Rollback()
+
+	result := tx.Delete(&models.AdvisoryAccountData{}, "systems_installable <= 0 AND systems_applicable <= 0")
+	if result.Error != nil {
+		return result.Error
+	}
+
+	tx.Commit()
+	utils.LogInfo("nDeleted", result.RowsAffected, "advisory_account_data cleaned successfully")
+	return nil
</code_context>
<issue_to_address>
**issue (bug_risk):** Avoid deferring Rollback after a successful Commit and check Commit errors

With `Begin` + `defer tx.Rollback()` + `tx.Commit()`, `Rollback` still runs after `Commit`, which can generate spurious errors/logs depending on the driver, and `Commit` errors are currently ignored.

Prefer an explicit pattern that:
- Checks `Begin` error
- Uses `defer` only for panic recovery
- Rolls back on intermediate errors
- Checks and returns the `Commit` error

For example:

```go
func CleanAdvisoryAccountData() error {
    tx := tasks.CancelableDB().Begin()
    if tx.Error != nil {
        return tx.Error
    }

    defer func() {
        if r := recover(); r != nil {
            tx.Rollback()
            panic(r)
        }
    }()

    result := tx.Delete(&models.AdvisoryAccountData{}, "systems_installable <= 0 AND systems_applicable <= 0")
    if result.Error != nil {
        tx.Rollback()
        return result.Error
    }

    if err := tx.Commit().Error; err != nil {
        return err
    }

    utils.LogInfo("nDeleted", result.RowsAffected, "advisory_account_data cleaned successfully")
    return nil
}
```

Same fix applies to `CleanAccountAdvisory()`.
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Comment thread tasks/cleaning/clean_account_advisory.go
Comment on lines +39 to +48
func CleanAdvisoryAccountData() error {
tx := tasks.CancelableDB().Begin()
defer tx.Rollback()

result := tx.Delete(&models.AdvisoryAccountData{}, "systems_installable <= 0 AND systems_applicable <= 0")
if result.Error != nil {
return result.Error
}

tx.Commit()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (bug_risk): Avoid deferring Rollback after a successful Commit and check Commit errors

With Begin + defer tx.Rollback() + tx.Commit(), Rollback still runs after Commit, which can generate spurious errors/logs depending on the driver, and Commit errors are currently ignored.

Prefer an explicit pattern that:

  • Checks Begin error
  • Uses defer only for panic recovery
  • Rolls back on intermediate errors
  • Checks and returns the Commit error

For example:

func CleanAdvisoryAccountData() error {
    tx := tasks.CancelableDB().Begin()
    if tx.Error != nil {
        return tx.Error
    }

    defer func() {
        if r := recover(); r != nil {
            tx.Rollback()
            panic(r)
        }
    }()

    result := tx.Delete(&models.AdvisoryAccountData{}, "systems_installable <= 0 AND systems_applicable <= 0")
    if result.Error != nil {
        tx.Rollback()
        return result.Error
    }

    if err := tx.Commit().Error; err != nil {
        return err
    }

    utils.LogInfo("nDeleted", result.RowsAffected, "advisory_account_data cleaned successfully")
    return nil
}

Same fix applies to CleanAccountAdvisory().

@katarinazaprazna
katarinazaprazna force-pushed the parallel-advisory-cleanup branch from 27e51dd to 82efcf3 Compare August 3, 2026 13:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants